Add Starlette Admin interface for Location management (MS Access replacement) by kbighorse · Pull Request #331 · DataIntegrationGroup/OcotilloAPI

kbighorse · 2026-01-01T22:44:17Z

Summary

This PR implements a web-based admin interface to replace MS Access for managing Location records during the AMPAPI → NMSampleLocations migration.

Key Features:

🔐 Authentication: Integrates with existing Authentik OIDC auth
📋 List View: MS Access Datasheet-style grid with sorting, filtering, search
✏️ Forms: Create/edit forms with WKT coordinate input
🔒 RBAC: Admin/Editor/Viewer permissions
📊 Export: CSV and Excel export (like MS Access export)
⚡ Bulk Actions: Publish/unpublish selected locations
🗺️ WKT Field: Custom field handler for PostGIS geometry

Context: MS Access → Starlette Admin Migration

Staff are accustomed to using MS Access to manage the AMPAPI SQL Server database. This PR provides equivalent functionality in a web-based interface.

What Staff Are Gaining

✅ Multi-user access - No more "database is locked" errors
✅ Remote access - Work from anywhere (no VPN/RDP required)
✅ No file corruption - PostgreSQL is more robust than .accdb files
✅ Better security - User-level permissions vs. file permissions
✅ Audit trail - Track who changed what when
✅ Cross-platform - Works on Mac/Linux/iPad, not just Windows

What Staff Are Losing

❌ Offline access - Requires internet connection
⚠️ Complex queries - Use filters instead of Access query designer
⚠️ Custom reports - Export to CSV → Excel instead

AMPAPI → NMSampleLocations Schema Changes

Primary Key: GUID → Auto-Increment Integer

AMPAPI (SQL Server):

LocationId = Column(UNIQUEIDENTIFIER, primary_key=True)

NMSampleLocations (PostgreSQL):

id: Mapped[int]  # Auto-increment primary key
nma_pk_location: Mapped[UUID]  # Legacy GUID preserved for traceability

Coordinates: UTM → WGS84 Point Geometry

AMPAPI stored UTM Zone 12N/13N coordinates in separate fields:

Easting: 385000
Northing: 3900000
UTMZone: 13

NMSampleLocations stores WGS84 coordinates as PostGIS POINT:

point: POINT(-106.123 35.456)  -- WGS84 (SRID 4326)

For Staff: Instead of entering Easting/Northing/UTM Zone, enter:

POINT(-106.123 35.456)

Note: Longitude first, then latitude (not lat/lon!)

Release Status: Boolean → Lexicon Term

AMPAPI:

PublicRelease = Column(Boolean)  # True/False
Confidential = Column(Boolean)   # True/False

NMSampleLocations:

release_status: Mapped[str]  # 'draft', 'published', 'archived', etc.

Migration Mapping:

PublicRelease=True → release_status='published'
PublicRelease=False → release_status='draft'

Elevation: Feet → Meters

AMPAPI:

Altitude = Column(Float)  # Feet above sea level

NMSampleLocations:

elevation: Mapped[float]  # Meters (NAVD88 vertical datum)

Conversion: Feet → meters during migration

Site Type & PointID: Moved to Thing Table

AMPAPI stored PointID and SiteType in Location table.

NMSampleLocations moved these to Thing table:

Location can have multiple things (wells, springs)
PointID is thing-specific, not location-specific
Many-to-many relationship via LocationThingAssociation

Admin Interface Details

List View (MS Access Datasheet Equivalent)

Columns Displayed:

Location ID
Description
County
State
Elevation (meters)
Quad Name
Release Status
Created At
Updated By

Filters:

County (dropdown)
State (dropdown)
Release Status (dropdown)
Elevation (range)
Created At (date range)

Search:

Description
County
State
Quad Name

Actions:

Export to CSV
Export to Excel
Publish selected locations
Unpublish selected locations

Form View (MS Access Form Equivalent)

Fields:

Description - Brief description
Coordinates (WKT) - Format: POINT(lon lat)
- Help text explains conversion from UTM
- Validates WKT format
Elevation - Meters (NAVD88)
County - New Mexico county
State - Default: "New Mexico"
Quad Name - USGS quadrangle
Location Notes - General notes
Coordinate Notes - Coordinate source/accuracy
Release Status - draft/published

Read-Only Fields (migration data):

AMPAPI Location ID (legacy GUID)
AMPAPI Date Created
AMPAPI Site Date

Permissions

Role	Create	Edit	Delete	View
Admin	✅	✅	✅	All records
Editor	✅	✅	❌	All records
Viewer	❌	❌	❌	Published only

Implementation Details

Files Added

admin/__init__.py - Package initialization
admin/auth.py - Authentik OIDC authentication provider
admin/config.py - Admin configuration
admin/fields.py - Custom WKT field for PostGIS geometry
admin/views.py - LocationAdmin view

Files Modified

main.py - Mount admin at /admin
pyproject.toml - Add starlette-admin dependency
uv.lock - Update lock file

Testing Checklist

Next Steps

After this PR:

✅ Local testing - Test with development database
✅ User acceptance testing - Demo to 2-3 staff members
✅ User guide - Create documentation for staff
✅ Training - 1-hour session for staff
⏭️ Add more models - Thing, Sample, Observation, etc.

Pull request overview

This PR implements a web-based Starlette Admin interface to replace MS Access for managing Location records during the AMPAPI → NMSampleLocations migration. The interface integrates with existing Authentik OIDC authentication and provides familiar MS Access-like functionality including list views, forms, filtering, and bulk operations.

Key Changes:

Added Starlette Admin dependency for web-based database management
Implemented OIDC authentication integration with Authentik
Created Location admin view with RBAC permissions (Admin/Editor/Viewer roles)

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Added starlette-admin[i18n] dependency for admin interface functionality
main.py	Mounted admin interface at /admin route
admin/init.py	Package initialization exporting create_admin function
admin/config.py	Admin configuration with Location view registration
admin/auth.py	Authentik OIDC authentication provider with role mapping
admin/fields.py	Custom WKT field handlers for PostGIS geometry columns
admin/views.py	Location model admin view with list/form/permission configuration

Copilot · 2026-01-01T22:44:45Z

+        Only admins can create new locations.
+
+        Returns:
+            bool: True if user has 'admin' role
+        """
+        user = getattr(request.state, "user", None)
+        if user is None:
+            return False
+        return "admin" in getattr(user, "roles", [])


The docstring states "Only admins can create new locations" but according to the PR description's permission table, both Admins and Editors should have create permissions. Either update the docstring to match the actual permission model or update the implementation to allow editors to create locations.

Suggested change

Only admins can create new locations.

Returns:

bool: True if user has 'admin' role

"""

user = getattr(request.state, "user", None)

if user is None:

return False

return "admin" in getattr(user, "roles", [])

Admins and editors can create new locations.

Returns:

bool: True if user has 'admin' or 'editor' role

"""

user = getattr(request.state, "user", None)

if user is None:

return False

roles = getattr(user, "roles", [])

return "admin" in roles or "editor" in roles

@smoke

Create comprehensive Gherkin/Behave feature files documenting admin interface behavior and acceptance criteria. Features: - authentication.feature: Auth/authz flows, RBAC, JWT verification • 11 scenarios covering admin/editor/viewer roles • Security tests (invalid tokens, expired tokens) • Development mode behavior - location_admin.feature: Location CRUD operations • 24 scenarios covering full admin lifecycle • List view: search, filter, sort, pagination, export • Forms: create/edit with WKT coordinate validation • Bulk actions: publish/unpublish • Data visibility by release status • Permission checks for all operations - README.md: Documentation for running and extending tests • Test setup instructions • Tag reference • Example test runs • Troubleshooting guide These feature files serve as both: 1. Living documentation of admin behavior 2. Acceptance criteria for implementation 3. Automated test specifications (when step definitions added) Tags enable targeted test runs: behave features/admin/ --tags=@smoke behave features/admin/ --tags=@rbac behave features/admin/ --tags=@bulk-actions Next step: Implement step definitions using Playwright/FastAPI TestClient 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Fixes CI failures: - Replaced Unicode smart quotes with ASCII quotes - admin/fields.py: Fixed bullet points in WKT help text - admin/views.py: Fixed all string literals with smart quotes This resolves: - SyntaxError in fields.py line 120 - UnicodeDecodeError in views.py (byte 0x92)

Copilot

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Copilot · 2026-01-01T22:59:17Z

+    Examples:
+      | role   | email                  | can_create | can_edit | can_delete | visibility   |
+      | Admin  | admin@nmbgmr.nmt.edu   | should     | should   | should     | should       |
+      | Editor | editor@nmbgmr.nmt.edu  | should     | should   | should not | should       |


The test expectation shows Editor can delete ('should' in can_delete column), but according to the permission table in the PR description and the can_delete implementation in admin/views.py, Editors should not be able to delete. This test data appears incorrect.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

jirhiker · 2026-01-02T17:27:18Z

There is a lot to discuss regarding this PR. First this concept is/was being explored via fastapi-admin. Second ocotillo has this "admin" functionality. Third, user research and experience have repeatedly revealed and reinforced that users do not want single table interfaces- this was a major complaint when users used NMAQuiferDev. Lastly, given that end users will not use it, and developers already have clients available for database management, who is the audience for this functionality? Developing a consensus around this will be important moving forward

kbighorse · 2026-01-02T17:38:32Z

@jirhiker Yes, this is part of a proposal we will bring to you very shortly. More to come.

jirhiker · 2026-01-02T17:56:43Z

There is considerable potential for this approach, and this appears to be a well-designed and clean implementation. However, the issue we have always run into is that ultimately we find ourselves having to write a ton of custom JS to implement the functionality the users (or even admins) really want (Perhaps this is more a function of NMAquifers' poor data model than an issue with the monolithic approach). The need for a bunch of custom JS/CSS/HTML has led us to move all Web GUI development to JS/React applications. The need for responsive, user-friendly, modern UIs eventually supersedes the benefits of a monolithic server-side application. So if there are clear guardrails on what the Admin UI is for and who should know about it/use it then I think this could be made a priority

kbighorse · 2026-01-02T18:02:43Z

does MS Access support such functionality?

jirhiker · 2026-01-02T19:24:29Z

I'm not sure which functionality you are referring to. However, I don't think that using ms access/ampapi as a baseline is valuable in this case. Limiting ourselves to what ms access did even temporarily does not seem a worthwhile goal. The previous workflow for data entry via ms access and nmaquiiferdev were highly compromised and bespoke because of both technical and culture issues. The Ms access-centric view also neglects the fact that a significant number of users interact with our data entirely independently of Ms access. Only a small number of users enter data (2-3) via Ms access, a slightly larger group may have used Ms access to view data, however the majority of users worked with our data either via the "interactive web map" or with arcgis via data exports (only a few users were capable and confident enough to us Ms access to generate data exports)

The `description` field was missing parentheses on `mapped_column`, causing a syntax error when the model was loaded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Extends starlette_admin's AdminUser with a roles list to enable role-based access control in admin views. The roles are populated from Authentik groups during authentication. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Refactor admin views into a package structure (admin/views/) and add MS Access-style admin interfaces for core data models: - ThingAdmin: Wells, springs, construction details - ObservationAdmin: Water level measurements - ContactAdmin: Well owners and managers - SensorAdmin: Equipment inventory - DeploymentAdmin: Equipment installation log Each view includes: - List view with sorting, filtering, search, pagination - Create/Edit forms with field labels and help text - RBAC: Admin (full), Editor (create/edit), Viewer (published only) - Bulk actions: Publish/Unpublish selected records - Export: CSV and Excel formats This replaces the single views.py file with a modular package structure for better maintainability as more views are added. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add 8 new equipment type mappings to sensor_transfer.py: Precip Collector, Soil Moisture, Weather Station, Camera, Weir, Snow Lysimeter, Tipping Bucket, Lysimeter - Add 4 missing sensor types to lexicon.json: Weather Station, Weir, Snow Lysimeter, Lysimeter This fixes 71 equipment records that were previously skipping due to unmapped EquipmentType values. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use PointID as the location description field to satisfy the NOT NULL constraint on location.description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

When TRANSFER_LIMIT was set to -1 (meaning "no limit"), the condition `if limit and i >= limit` evaluated to True on first iteration because -1 is truthy and 0 >= -1. This caused transfers to exit immediately without processing any records. Changed condition to `if limit > 0 and i >= limit` (or with explicit None check) so that -1 and 0 correctly mean "no limit". 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Different wells (things) can have observation blocks with the same review_status, parameter_id, and overlapping time ranges. The previous unique constraint on (review_status, parameter_id, start_datetime, end_datetime) caused duplicate key errors. Changes: - Add thing_id foreign key to TransducerObservationBlock - Add Thing relationship for navigation - Update unique constraint to include thing_id - Update transfer code to extract thing_id from deployment before creating blocks 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Legacy data from MS Access may not have descriptions for all locations. Making this field nullable allows the transfer to proceed without requiring a description value. Changes: - Update Location model to set nullable=True on description - Add migration to alter the column constraint 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Configure SQLAlchemy connection pooling with configurable pool_size and max_overflow settings via environment variables. This enables concurrent database connections required for parallel transfer operations. - Add DB_POOL_SIZE and DB_MAX_OVERFLOW env vars (defaults: 10, 20) - Enable pool_pre_ping to verify connections before use - Apply to both CloudSQL and local PostgreSQL engines 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Restructure transfer pipeline to execute independent transfers concurrently using ThreadPoolExecutor. Transfers are organized into phases: Phase 1: Foundational (AquiferSystems, GeologicFormations) - parallel Phase 2: Wells - supports parallel mode via TRANSFER_PARALLEL_WELLS Phase 3: Group 1 (Screens, Contacts, WaterLevels, etc.) - parallel Phase 4: Sensors - sequential (required before continuous water levels) Phase 5: Group 2 (Pressure, Acoustic) - parallel Add helper functions for thread-safe transfer execution with timing. Retain sequential mode via TRANSFER_PARALLEL=0 for compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Implement row-level parallelization for wells transfer using batch processing with ThreadPoolExecutor. Key optimizations: - transfer_parallel(): Splits wells into batches across workers - _step_parallel_complete(): Creates well + ALL dependent objects (notes, status, provenances, measuring points, formation zone) in a single pass, eliminating the sequential after_hook bottleneck - Thread-safe aquifer handling with minimal lock contention - Pre-load formations per batch to avoid race conditions Performance: 9,887 wells in ~2 minutes (vs ~56 minutes with after_hook) Throughput: ~21 wells/sec with 4 workers Enable via TRANSFER_PARALLEL_WELLS=1, configure workers with TRANSFER_WORKERS (default: 4). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…hod for compatibility

jirhiker · 2026-01-06T06:30:48Z

Right now there is a lot of duplicated code in the ModelView subclasses. It would be great to consolidate functionality into a OcotilloModelView subclass. The duplicated code im referring to is stuff like this

#========== Permissions (RBAC) ==========

    def can_create(self, request: Request) -> bool:
        user = getattr(request.state, "user", None)
        if user is None:
            return False
        return "admin" in getattr(user, "roles", [])

    def can_edit(self, request: Request) -> bool:
        user = getattr(request.state, "user", None)
        if user is None:
            return False
        roles = getattr(user, "roles", [])
        return "admin" in roles or "editor" in roles

    def can_delete(self, request: Request) -> bool:
        user = getattr(request.state, "user", None)
        if user is None:
            return False
        return "admin" in getattr(user, "roles", [])

    def can_view_details(self, request: Request) -> bool:
        user = getattr(request.state, "user", None)
        return user is not None

    # ========== Data Visibility (Release Status Filter) ==========

    async def get_list_query(self, request: Request):
        query = select(self.model)

        user = getattr(request.state, "user", None)
        if user is None:
            return query.where(self.model.id == -1)

        roles = getattr(user, "roles", [])
        if "admin" in roles or "editor" in roles:
            return query
        else:
            return query.where(self.model.release_status == "published")

…tion

…ion-ms-access-jir Feature/admin location ms access jir

…ation

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…taging abd testing

…ed context

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…leaner configuration

…eamline upgrade process

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…, Data Provenance, Notes, Sample, and Group models

kbighorse and others added 4 commits January 1, 2026 14:38

Copilot AI review requested due to automatic review settings January 1, 2026 22:44

Copilot AI reviewed Jan 1, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings January 1, 2026 22:58

kbighorse and others added 2 commits January 1, 2026 14:59

Formatting changes

e94ad31

Copilot AI reviewed Jan 1, 2026

View reviewed changes

Copilot AI reviewed Jan 1, 2026

kbighorse and others added 12 commits January 2, 2026 20:51

Add location description from PointID in transfer

e2d267a

Use PointID as the location description field to satisfy the NOT NULL constraint on location.description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Formatting changes

2b38862

Copilot AI review requested due to automatic review settings January 5, 2026 17:03

Copilot AI reviewed Jan 5, 2026

jirhiker and others added 2 commits January 5, 2026 19:28

feat: add session middleware for admin auth flow and update login met…

dc0f3da

…hod for compatibility

Formatting changes

7fe5956

jirhiker and others added 6 commits January 6, 2026 11:16

feat: update pandas-stubs version to 2.3.3 and adjust version specifier

965133d

feat: change default value of TRANSFER_ASSETS to False

fa458c6

feat: enforce SESSION_SECRET_KEY environment variable requirement

39915e8

feat: add SESSION_SECRET_KEY to .env.example for middleware configura…

090c426

…tion

Merge pull request #335 from DataIntegrationGroup/feature/admin-locat…

0c47a12

…ion-ms-access-jir Feature/admin location ms access jir

feat: add Cloud SQL support and Alembic migration step in CI configur…

9e9eada

…ation

Copilot AI review requested due to automatic review settings January 6, 2026 21:05

Copilot AI reviewed Jan 6, 2026

jirhiker added 2 commits January 6, 2026 14:22

feat: add SESSION_SECRET_KEY to CI configuration for production and s…

e4d4252

…taging abd testing

feat: update add_block function to include thing parameter for enhanc…

c617ac8

…ed context

Copilot AI review requested due to automatic review settings January 6, 2026 21:28

Copilot AI reviewed Jan 6, 2026

jirhiker added 2 commits January 6, 2026 14:36

feat: remove GOOGLE_APPLICATION_CREDENTIALS from CD_staging.yml for c…

eabc491

…leaner configuration

feat: refactor initial migration to remove commented-out code and str…

947aac0

…eamline upgrade process

Copilot AI review requested due to automatic review settings January 6, 2026 22:56

Copilot AI reviewed Jan 6, 2026

jirhiker and others added 2 commits January 6, 2026 22:56

Formatting changes

1817c7f

Merge branch 'staging' into feature/admin-location-ms-access

bde8531

Copilot AI review requested due to automatic review settings January 7, 2026 04:42

Copilot AI reviewed Jan 7, 2026

feat: add admin views for Lexicon, Aquifer, Asset, Geologic Formation…

bbc82f8

…, Data Provenance, Notes, Sample, and Group models

jirhiker merged commit 803f1d1 into staging Jan 7, 2026
6 checks passed

Uh oh!

Conversation

kbighorse commented Jan 1, 2026

Summary

Context: MS Access → Starlette Admin Migration

What Staff Are Gaining

What Staff Are Losing

AMPAPI → NMSampleLocations Schema Changes

Primary Key: GUID → Auto-Increment Integer

Coordinates: UTM → WGS84 Point Geometry

Release Status: Boolean → Lexicon Term

Elevation: Feet → Meters

Site Type & PointID: Moved to Thing Table

Admin Interface Details

List View (MS Access Datasheet Equivalent)

Form View (MS Access Form Equivalent)

Permissions

Implementation Details

Files Added

Files Modified

Testing Checklist

Next Steps

Related Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jirhiker commented Jan 2, 2026

Uh oh!

kbighorse commented Jan 2, 2026

Uh oh!

jirhiker commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kbighorse commented Jan 2, 2026

Uh oh!

jirhiker commented Jan 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jirhiker commented Jan 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jirhiker commented Jan 2, 2026 •

edited

Loading